• Defining text mining
• Text mining first steps
• Tokenisation
• Lemmatisation
• Stemming
• tf-idf

“[…] text mining seeks to extract useful information from data sources through the identification and exploration of interesting patterns […]“

“[…] data sources are document collections, and interesting patterns are found not among formalized database records but in the unstructured textual data in the documents in these collections” (Feldman and Sanger 2006)

“Text mining represents the ability to take large amounts of unstructured language and quickly extract useful and novel insights that can affect stakeholder decision‐making“ (Kwartler 2017)

• Increasing access to data in the form of text
  • Newspapers
  • Bibliometric data (full text of publications and patents)
  • Social media (e.g. Twitter)
  • Parliamentary debates (e.g. https://hansard.parliament.uk)
  • …
• A phenomenon of considerable magnitude
  (https://www.webfx.com/internet-real-time/)
• Textual data are unstructured

• Bag of words
  • Words or groups of words are considered to be a feature of documents
  • The order of words and the grammar are not considered
  • Computationally inexpensive and data ready for machine learning (Document-Term Matrices)

• Syntactic parsing
  • Syntactic rules used to build the sentence are defined
  • Words or groups of words are tagged (e.g. adjectives, nouns, verbs)
  • Computationally expensive and complex language-dependent models
  • In-depth analysis of the relationships between the elements of a corpus

Source: The Guardian, 12 November 2021

Wickham and Grolemund (2017) describe tidy data as data were

• Observations are in rows
• Variables are in columns
• Each value is in a cell

• Text data may be in formats that are not ready for text mining
• These include:
  • PDF files
  • XML files
  • scanned images
  • recorded audio (e.g. speeches)
  • www
  • handwritten text
  • …

• A key preparatory task is perceptual grouping
• The aim is to group the primitive elements of the documents into objects of higher levels, i.e. to generate an O-Tree
• For example, some OCR software packages can recognise objects such as columns from scanned images

• Natural Language Processing (NLP) is an important area of an interdisciplinary research domain called computational linguistic
• NLP provides techniques to transform and process text data, so to identify patterns in these data
• NLP is particulalrly important for “syntactic parsing” than for “bag of words” text mining
• Three main approaches:
  1. Part-of-Speech (POS) tagging
  2. Full parsing
  3. Shallow parsing

Part-of-Speech (POS) tagging

• Words are categorised according to the role they play in the sentence: article, noun, verb, adjective, preposition, number, proper noun, etc.
• List of POS tags
• The tokenisation of text into words or groups of words (we will see this in practice later)

Full parsing

• The objective is to perform a full syntactical analysis of sentence identifying two elements

  • Constituency grammars: short phases that convey a meaning
    • Noun phrase (NP): subjects or objects to a verb
    • Verb phrase (VP): verbs
    • Adjective phrase (ADJP): adjectives to qualify nouns and pronouns
    • Adverb phrase (ADVP): adverbs to modify nouns, verbs, or adverbs
    • Prepositional phrase (PP): prepositions to describe words or phrases
  • Dependency grammars: relationships between words (e.g. a subject and an object depend on a verb)

Shallow parsing

• Performing a full parsing for large text corpora could be computationally expensive
• Shallow parsing reduces the depth of the parsing analysis for the sake of speed
  • Simple and short phrases are identified
  • Unclear and ambiguous dependency are left unresolved
• The results of the shallow parsing are sufficient to characterise a corpus

• Text categorization or classification
  • Assigning tags representing concepts or keywords to documents
  • Tags may have a hierarchical structure
  • Examples: Medical Subject Headings, JEL Classification System

• Information extraction
  • Extracting information and presenting this in a structured format (e.g. tables, charts), beyond the simple information retrieval (e.g. identification of keywords in text)
  • Entity extraction: entities in the text (e.g. individuals)
  • Relation extraction: relationships between entities as in the text
  • Coreference resolution: expressions referring to the same entity (e.g. Daniel, him, his)

• We will focus on text mining based on bags of words
• Techniques to process and analyse the text
  • Tokenisation
  • Lemmatisation
  • Stemming
  • tf-idf
  • Visualisation
  • Sentiment analysis
  • Topic modelling

• The objective is to break text into meaningful elements
• For example, we can break a document into
  • chapters
  • sections
  • paragraphs
  • sentences
  • words
  • syllables

• A commonly used approach is to break the text into words, i.e. tokens
• A token could be
  • a single word (unigram)
  • two subsequent words (i.e. bigram)
  • …
  • n subsequent words (n-gram)
• We can use token to build a tidy dataset, where each row reports a token

• In tidytext, the function unnest_tokens enables us to tokenise the text
• This function
  • removes punctuation
  • converts tokens into lower case
• For more options see ??unnest_tokens

• Text from 692 news English articles on COP26 published by newspapers on 12 November 2021

• Each news article has a numeric id
• Text is include in the fields
  • “Title”
  • “Headline”
  • “Hlead”

• We load the readr, tidyverse, tidytext, and ggplot packages
• We store our data into R as a dataframe/tibble
• We focus on the text reported in the field “Title”

library(readr)
library(ggplot2)
library(tidyverse)
library(tidytext)

my_text <- read_csv("news_articles_example.csv") %>%
  select(id, Title)

print(my_text, n = 6)

## # A tibble: 692 × 2
##      id Title                                                                   
##   <dbl> <chr>                                                                   
## 1     1 The unreal spectacle of COP26;COP26 has had a prepackaged feel to it, a…
## 2     2 Climate activists send a message for COP26                              
## 3     3 COP26: Nicola Sturgeon urges Boris Johnson to return and use position t…
## 4     4 COP26: Nicola Sturgeon urges Boris Johnson to return and use position t…
## 5     5 COP26: Police Scotland arrested eight people on penultimate day of Glas…
## 6     6 COP26: Top 10 bizzare moments of the Glasgow climate talks ranked       
## # … with 686 more rows

We tokenise each sentence into unigrams

my_text_uni <- my_text %>%
  unnest_tokens(output = word, input = Title)

print(my_text_uni, n = 6)

## # A tibble: 7,925 × 2
##      id word     
##   <dbl> <chr>    
## 1     1 the      
## 2     1 unreal   
## 3     1 spectacle
## 4     1 of       
## 5     1 cop26    
## 6     1 cop26    
## # … with 7,919 more rows

We could also tokenise into n-grams, for example bigrams

my_text_bi <- my_text %>%
  unnest_tokens(output = bigram, input = Title, 
                token = "ngrams", n = 2)

print(my_text_bi, n = 6)

## # A tibble: 7,238 × 2
##      id bigram          
##   <dbl> <chr>           
## 1     1 the unreal      
## 2     1 unreal spectacle
## 3     1 spectacle of    
## 4     1 of cop26        
## 5     1 cop26 cop26     
## 6     1 cop26 has       
## # … with 7,232 more rows

… or trigrams

my_text_tri <- my_text %>%
  unnest_tokens(output = trigram, input = Title, 
                token = "ngrams", n = 3)

print(my_text_tri, n = 6)

## # A tibble: 6,557 × 2
##      id trigram             
##   <dbl> <chr>               
## 1     1 the unreal spectacle
## 2     1 unreal spectacle of 
## 3     1 spectacle of cop26  
## 4     1 of cop26 cop26      
## 5     1 cop26 cop26 has     
## 6     1 cop26 has had       
## # … with 6,551 more rows

… or sequences of characters (five in the case below)

my_text_char <- my_text %>%
  unnest_tokens(output = character_shingles, input = Title,  
                token = "character_shingles", n = 5)

print(my_text_char, n = 6)

## # A tibble: 36,769 × 2
##      id character_shingles
##   <dbl> <chr>             
## 1     1 theun             
## 2     1 heunr             
## 3     1 eunre             
## 4     1 unrea             
## 5     1 nreal             
## 6     1 reals             
## # … with 36,763 more rows

my_text_uni_pos <- my_text_uni %>%
  left_join(parts_of_speech, by  = "word")

print(my_text_uni_pos, n = 6)

## # A tibble: 15,047 × 3
##      id word      pos             
##   <dbl> <chr>     <chr>           
## 1     1 the       Definite Article
## 2     1 the       Adverb          
## 3     1 unreal    Adjective       
## 4     1 spectacle Noun            
## 5     1 of        Noun            
## 6     1 of        Preposition     
## # … with 15,041 more rows

my_text_uni_pos <- my_text_uni_pos %>%
  group_by(pos) %>%
  count(pos, sort = T)

print(my_text_uni_pos, n = 6)

## # A tibble: 13 × 2
## # Groups:   pos [13]
##   pos                       n
##   <chr>                 <int>
## 1 Noun                   4953
## 2 <NA>                   1521
## 3 Adverb                 1456
## 4 Verb (usu participle)  1449
## 5 Preposition            1395
## 6 Adjective              1175
## # … with 7 more rows

Let’s focus on unigrams and count the number of words

my_text_uni_count <- my_text_uni %>% 
  count(word, sort = T)

print(my_text_uni_count, n = 6)

## # A tibble: 2,436 × 2
##   word        n
##   <chr>   <int>
## 1 to        300
## 2 the       188
## 3 of        161
## 4 cop26     160
## 5 on        148
## 6 climate   135
## # … with 2,430 more rows

Basic descriptive statistics

summarise(my_text_uni_count, 
          num_words = n(),
          mean = mean(n),
          sd   = sd(n),
          min  = min(n),
          max  = max(n))

## # A tibble: 1 × 5
##   num_words  mean    sd   min   max
##       <int> <dbl> <dbl> <int> <int>
## 1      2436  3.25  11.1     1   300

We can plot the most frequent words representing the 1% of the total sample

min_occur <- quantile(my_text_uni_count$n, 0.99)

g <- my_text_uni_count %>%
  filter(n >= min_occur) %>%
  mutate(word = reorder(word, n)) %>% 
  ggplot(aes(x = word, y = n, fill = word)) +
  geom_col() +
  theme(legend.position = "none") +
  coord_flip()

• Many words are not particularly relevant (e.g. to, the, in)
• These terms are called stopwords

data(stop_words)

my_text_uni <- my_text_uni %>%
      anti_join(stop_words)

my_text_uni_count <- my_text_uni %>% 
  count(word, sort = T)

g <- my_text_uni_count %>%
  filter(n >= min_occur) %>%
  mutate(word = reorder(word, n)) %>% 
  ggplot(aes(x = word, y = n, fill = word)) +
  geom_col() +
  theme(legend.position = "none") +
  coord_flip()

• Numbers my not be meaningful for your analysis
• We can remove numbers converting text into numeric format

my_text_uni <- my_text_uni %>%
    mutate(word_numeric = as.numeric(word))

## Warning in mask$eval_all_mutate(quo): NAs introduced by coercion

• NAs in the table are word, so we can filter those only
• We can reduce the threshold to the most frequnet words representing the 2% of the total sample so to explore more words

my_text_uni <- my_text_uni %>%
  filter(is.na(word_numeric))

my_text_uni_count <- my_text_uni %>% 
  count(word, sort = T)

min_occur <- quantile(my_text_uni_count$n, 0.98)

g <- my_text_uni_count %>%
  filter(n >= min_occur) %>%
  mutate(word = reorder(word, n)) %>% 
  ggplot(aes(x = word, y = n, fill = word)) +
  geom_col() +
  theme(legend.position = "none") +
  coord_flip()

• In some case, you may want to remove words that are not particularly relevant

my_stop_words <- tibble(word = c("cop26", "final", "news"), 
                        lexicon = "mywords")

my_text_uni<- my_text_uni %>%
      anti_join(my_stop_words)

g <- my_text_uni %>% 
  count(word, sort = T) %>%
  filter(n >= min_occur) %>%
  mutate(word = reorder(word, n)) %>% 
  ggplot(aes(x = word, y = n, fill = word)) +
  geom_col() +
  theme(legend.position = "none") +
  coord_flip()

• Google Books provides access to tokenised data for several millions of books
• Data structure
  • token (n-gram)
  • number of occurrences (match_count)
  • number of occurrences in distinct books (volume_count)
• Warning: The database includes a large amount of data (several gigabytes)
• Google Books query interface
• We can use the ngramr package to gather data

library(ngramr)
library(ggplot2)
googlebook_data  <- ngram(c("science", "technology", "innovation"), 
             smoothing = 0, year_start = 1900)
g <- ggplot(googlebook_data, 
       aes(x = Year, y = Frequency, colour=  Phrase)) +
  geom_line()

• Individual tokens are reduced to their base form, called “lemma”
  • “am,” “are,” “is” –> “be”
  • “cars” –> “car”
  • “arrived” –> “arrive”
  • …
• Lemmatisation normalise text (e.g. counting)
• This processing is useful for sentiment analysis

• We can use the package textstem
• We have already remove stopwords, numbers, and additional stopwords from the list of unigrams

library(textstem)
my_text_uni <- my_text_uni %>%
  mutate(word_lemma = textstem::lemmatize_words(word))

• We can now count lemmas and plot the results

my_text_uni_count <- my_text_uni %>% 
  count(word_lemma, sort = T)

min_occur <- quantile(my_text_uni_count$n, 0.95)

g <- my_text_uni_count %>%
  filter(n >= min_occur) %>%
  mutate(word_lemma = reorder(word_lemma, n)) %>% 
  ggplot(aes(x = word_lemma, y = n, fill = word_lemma)) +
  geom_col() +
  theme(legend.position = "none") +
  coord_flip()

• Stemming is similar to the lemmatisations
• It removes the suffix of a word
  • “playing” –> “play”
  • “consultant” –> “consult”
  • “magically” –> “magic”
  • …

• We can use the package SnowballC
• We have already removed stopwords, numbers, and additional stopwords from the list of unigrams

library(SnowballC)
my_text_uni <- my_text_uni %>%
  mutate(word_stem = wordStem(word))

• We can now count words/stems and plot the results

my_text_uni_count <- my_text_uni %>% 
  count(word_stem, sort = T)

min_occur <- quantile(my_text_uni_count$n, 0.95)

g <- my_text_uni_count %>%
  filter(n >= min_occur) %>%
  mutate(word_stem = reorder(word_stem, n)) %>% 
  ggplot(aes(x = word_stem, y = n, fill = word_stem)) +
  geom_col() +
  theme(legend.position = "none") +
  coord_flip()

• Term frequency provides an indication of the most frequent words in a corpus
• Some words may be not very frequent in a corpus, but they may be particular for some documents in the corpus
• We can identify these words using the term frequency-inverse document frequency indicator

• We use the lead paragraph (“Hlead”), we tokenize data into unigram, we remove stopwords and numbers
• Our corpus include 242,831 id-unigram pairs

my_text_uni<- read_csv("news_articles_example.csv") %>%
  select(id, Hlead) %>%
  unnest_tokens(output = word, input = Hlead) %>%
  anti_join(stop_words) %>%
  mutate(word_numeric = as.numeric(word)) %>%
  filter(is.na(word_numeric)) %>%
  select(-word_numeric)

## Warning in mask$eval_all_mutate(quo): NAs introduced by coercion

We can calculate the tf (term frequency by document)

article_words <- my_text_uni %>%
  count(id, word, sort = T) %>% 
  ungroup()

head(article_words)

## # A tibble: 6 × 3
##      id word        n
##   <dbl> <chr>   <int>
## 1   659 pwa        84
## 2   659 editor     70
## 3   659 bar        66
## 4   170 water      57
## 5   124 parties    52
## 6   124 paris      50

total_words <- article_words %>%
      group_by(id) %>%
      summarize(total = sum(n))

print(total_words, n = 6)

## # A tibble: 692 × 2
##      id total
##   <dbl> <int>
## 1     1   381
## 2     2   378
## 3     3   558
## 4     4   558
## 5     5   346
## 6     6   722
## # … with 686 more rows

We remove document with only a single word

article_words <- left_join(article_words, total_words) %>%
  filter(total > 1) %>%
  mutate(word_freq = n/total)

print(article_words, n = 6)

## # A tibble: 131,673 × 5
##      id word        n total word_freq
##   <dbl> <chr>   <int> <int>     <dbl>
## 1   659 pwa        84  1168    0.0719
## 2   659 editor     70  1168    0.0599
## 3   659 bar        66  1168    0.0565
## 4   170 water      57  1053    0.0541
## 5   124 parties    52  2040    0.0255
## 6   124 paris      50  2040    0.0245
## # … with 131,667 more rows

Let’s look at the term frequency distribution of three random articles

g <-  article_words %>%
  filter(id == 107 | id == 175 | id == 288 | id == 631)  %>%
  ggplot(aes(word_freq, fill = as.character(id))) + 
  geom_histogram() + 
  xlim(NA, 0.05) +
  theme(legend.position = "none") +
  facet_wrap(~id, ncol = 4)

• Some words are very frequent in a number of documents (articles in this case)
• tf-idf helps us to identify words that are able to characterize documents
• In tidytext the function bind_tf_idf calculate the tf-idf scores starting from a tidy text dataset: one row per token per document including the information about term frequency
• Check the help guide ??bind_tf_idf

Let’s start from reading the data again

article_words_tfidf <- read_csv("news_articles_example.csv") %>%
  select(id, Hlead) %>%
  unnest_tokens(output = word, input = Hlead) %>%
  anti_join(stop_words) %>%
  mutate(word_numeric = as.numeric(word)) %>%
  filter(is.na(word_numeric)) %>%
  select(-word_numeric) %>%
  count(id, word, sort = T) %>% 
  ungroup() %>%
  bind_tf_idf(word, id, n) 

## Warning in mask$eval_all_mutate(quo): NAs introduced by coercion

print(article_words_tfidf, n = 6)

## # A tibble: 131,673 × 6
##      id word        n     tf   idf tf_idf
##   <dbl> <chr>   <int>  <dbl> <dbl>  <dbl>
## 1   659 pwa        84 0.0719  6.54 0.470 
## 2   659 editor     70 0.0599  2.93 0.176 
## 3   659 bar        66 0.0565  4.14 0.234 
## 4   170 water      57 0.0541  2.29 0.124 
## 5   124 parties    52 0.0255  2.26 0.0577
## 6   124 paris      50 0.0245  1.56 0.0381
## # … with 131,667 more rows

Let’s look at top 15 words by tf-idf in the case of four random articles

g <- article_words_tfidf %>%
  filter(id == 107 | id == 175 | id == 288 | id == 631)  %>%
  group_by(id) %>%
  top_n(15, tf_idf) %>%
  ungroup %>%
  mutate(id = as.factor(id),
         word = reorder_within(word, tf_idf, id)) %>%
  ggplot(aes(word, tf_idf, fill = id)) +
  geom_col() +
  facet_wrap(~id,  nrow = 2, scales = "free_y") +
  coord_flip() +
  theme(legend.position = "none") +
  scale_x_reordered()

• In the case of bigrams, to remove stopwords you need first to separate(), filter stop words, and then unite() bigrams again
• Similarly in the case of n-grams

article_words_tfidf <- read_csv("news_articles_example.csv") %>%
  select(id, Hlead) %>%
  unnest_tokens(output = bigram, input = Hlead, 
                token = "ngrams", n = 2) %>%
  separate(bigram, c("word_1", "word_2"), sep = " ") %>%
  filter(!word_1 %in% stop_words$word) %>%
  filter(!word_2 %in% stop_words$word) %>%
  unite(bigram, word_1, word_2, sep = " ") %>%
  count(id, bigram, sort = T) %>% 
  bind_tf_idf(bigram, id, n) 

print(article_words_tfidf, n = 6)

## # A tibble: 73,776 × 6
##      id bigram              n     tf   idf tf_idf
##   <dbl> <chr>           <int>  <dbl> <dbl>  <dbl>
## 1   659 editor bar         64 0.0706  6.54 0.461 
## 2   124 paris agreement    44 0.0486  1.89 0.0917
## 3   659 pwa editor         34 0.0375  6.54 0.245 
## 4   659 data pwa           32 0.0353  6.54 0.231 
## 5   659 bar panel          30 0.0331  6.54 0.216 
## 6   248 words photos       20 0.0430  6.54 0.281 
## # … with 73,770 more rows

Let’s look at top 15 bigrams by tf-idf in the case of four random articles

g <- article_words_tfidf %>%
  filter(id == 107 | id == 175 | id == 288 | id == 631) %>%
  group_by(id) %>%
  top_n(5, tf_idf) %>%
  ungroup %>%
  mutate(id = as.factor(id),
         bigram = reorder_within(bigram, tf_idf, id)) %>%
  ggplot(aes(bigram, tf_idf, fill = id)) +
  geom_col() +
  facet_wrap(~id,  nrow = 2, scales = "free_y") +
  coord_flip() +
  theme(legend.position = "none") +
  scale_x_reordered()

1. Students are grouped in randomly generated groups
2. Groups will select at least one text corpus and develop a script in R that undertake a text mining analysis (data are available on Canvas)
3. Groups will upload the R script and the main findings in Padlet by the end of Week 9 workshop

Group 1
Adebisi, Jongho, Maria, Keiho

Group 2
Charunan, Poojani, Abdul, Satoshi

Group 3
Oscar, Tsukumo, Jiyoung, Nicholas

Group 4
Alessandro, Shaunna, Jonathan, Rachel

• Visualising text data
• Sentiment analysis
• Regular expressions
• Introduction to topic modelling